Recovery Time Objective (RTO) is the maximum downtime an organization will accept after a failure. Your actual recovery time is the sum of every step between the failure and users working again: detecting the problem, responding, restoring data, reconnecting applications, and validating the result. Most RTOs fail because they only count the restore.
Most organizations struggle to set an RTO that’s both meaningful and realistic, and even those that do often find their processes don’t support it. In this article, I’ll walk through nine common reasons recovery takes longer than expected – and the actions you should take as a result.
What makes up your real recovery time?
TL;DR: Real recovery time = detection + response + restore or failover + application reconnection + validation
It’s a question that no one really wants to ask or talk about.
“What’s the maximum amount of downtime that will occur in the event of a failure?”
The desired number, of course, is zero, but reality and cost quickly render that unrealistic. An initial discussion can determine RTO based on hardware and software investment, but that’s only the beginning of the RTO calculation.
Ultimately, without proper training, documentation, process, and testing, the number itself is nothing more than a wild guess. So much can go wrong, and it’s very easy for an organization to become complacent in the absence of outages or hardware/software failures.
“It’s never happened before” becomes a silent justification for inaction – and the process to detect and respond to an outage is non-trivial in time and resource needs.
RTO is a combination of many processes, including the, “uh oh, is something wrong?!” step. I’ll split this article up into multiple sub-steps, working through problems within each one.
Slow detection: how monitoring gaps inflate RTO
Knowing about outages requires alerting, monitoring, and meaningful metrics. Outages are quite often not caused by a single immediate incident; instead, they are usually the culmination of a sequence of events.
For example, imagine the following situation:
- A software release is deployed to production.
- Included in the release is a single (very) poorly-written query.
- The query consumes excessive CPU and memory, taking many minutes to execute.
- CPU increases over the next month as new features are adopted.
- CPU on one server maxes out – causing long delays, timeouts, and errors.
- After 4 hours of this behavior, the server crashes.
- Upon server restart, a critical database is corrupt and unavailable.
How did this happen?
This is a train wreck happening right in front of our eyes – but how exactly did it happen? Well, the true failure here is that nobody noticed the unusually high CPU after the release.
Had this been caught, there would have been time to identify the inefficient query, optimize it, and deploy a fix before things spiraled out of control like they did – ultimately leading to the corrupted database.
RTO is 100% dependent on effective monitoring and alerting. No recovery time can be promised unless the time to identify a problem can be estimated. Once a problem is found, there needs to be a clear process to triage it and ensure that the best people are available to tackle it.
All of this points to observability. A data platform and any apps that use it must have the necessary tooling in place to allow developers and engineers to know (promptly) when problems occur.
Equally important is the ability to review performance history and capacity-plan for the future. It’s far better to expand resources on your own schedule than to do so when servers are on fire at 2am!
One very important note on metrics: the time between data samples for metrics like CPU, blocking, or a downed service is added to RTO.
For example, if metrics are captured and analyzed automatically every 2 minutes, then it can take up to 2 minutes for a problem to present itself – and that number must be factored into an RTO estimate.
The larger and more complex an application and its data, the more expansive these monitoring metrics will be – and RTO depends on this. Effective monitoring and alerting ensures that incidents are not prolonged due to poor response time.
Future-proof database monitoring with Redgate Monitor
Response delays: how long until someone starts working?
Timely alerting on incidents is important, but equally critical is the need to respond effectively.
Who is responsible for an incident response team? And, in smaller organizations, who is the person on-call and watching out for trouble?
At its core, this is simple. If an incident occurs, the amount of time needed for a person to begin working on it is added to RTO.
So, if it takes an average of 5 minutes for an operator to sign on and begin assessing the situation, add that 5 minutes to the RTO estimate!
How automation can speed up the response
Automation can greatly speed up the response by:
- Taking simple actions in direct response to a problem.
- Packaging up the incident and its associated metrics into an easily-consumable form.
- Performing actions that a human would do anyway.
- Allocating resources if the problem is directly related to limited resources.
Whatever this response looks like, it must be documented and understood by those responsible for it. Otherwise, an incident will lead to inaction, confusion or – worse still – the wrong response.
Common causes of slow recovery: a readiness checklist
In the world of data platform disasters, there are a set of common incidents that lead to trouble far more often than others. The following is a checklist to essentially allow you to answer the question: “are we ready for this?”
This list is by no means complete but has many of the top gotchas that create long, late, miserable nights for software developers.
Untested backups and slow restores
Does your organization have backups that run effectively and often enough?
FULLbackups periodically – ensuring that a database can quickly be restored in its entirety when needed.DIFFERENTIAL(incremental) backups that run on days when full backups are not run.TRANSACTION LOGbackups to ensure point-in-time recovery.- The RETENTION of backups that meets the needs of your organization. Are backups stored in multiple locations to ensure they are safe from data loss?
Equally importantly: do you actually test your backups?
This is one of the most critical challenges for organizations. It’s seen as success if backups are taken regularly but, if nobody actually tests the restoring of those backups, do they really work? And do those tests simulate what a real disaster would look like?
Additionally, are there scripts or automations available that allow for speedy restore operations? You don’t want to have to run SQL queries by hand in an emergency, with the CTO (Chief Technology Officer) staring over your shoulder!
Remember, also, that restore times will add to RTO. This becomes even more critical if your organization hosts very large databases, or a large number of databases.
How long does it take to restore a 50TB database? How long does it take to restore 5,000 databases? Are processes available that can quickly assemble RESTORE scripts for you? Does storage IO and network bandwidth provide the needed resources to restore lost data fast enough?
Finally, do those systems need to be temporarily allocated extra hardware while recovery is in progress to ensure ample speed?
These are all questions you need to have an answer for.
No high availability: single points of failure and failover time
High availability (HA) solutions turn disasters into routine events – but not all organizations have these in place. It’s important to evaluate what HA solutions make sense for your organization, answering the following questions in the process:
- What are the single points of failure for my applications?
- What are the most likely failure points in our infrastructure?
- If any infrastructure component fails, what can be automated up-front to reduce downtime?
- Can servers automatically fail over to replicas when needed?
These types of solutions not only help in emergencies, but assist in speeding up downtime due to planned maintenance and patching. I can’t count how many times I’ve listened to stories of engineers waiting for software patches to complete while an app was down!
So, being able to patch a passive replica, fail over to it, and then patch the previously active replica, can prevent significant downtime – and stress!
Note that, if cost is a significant issue, high availability solutions do not need to be permanent. They can be created quickly and used when risk is highest, or during maintenance events. They can then be switched off as passive standby hardware if the need arises again. While imperfect, this is far better than no HA at all.
While typically small, the time needed for HA solutions to fail over and become active is a component of RTO.
Application dependencies: connection strings, DNS, and config changes
When we think of high availability, disaster recovery, and RTO, we most often think of hardware – but how an application handles a disaster is just as critical.
If changes occur in infrastructure, for example, can the application be updated quickly? And can those changes be scripted or automated to be as fast as possible?
Such changes include:
- Connection string changes
- DNS changes
- Application metadata updates
- Monitoring updates
This is often overlooked, but making application configuration updates manually after hardware changes can be a huge time-waster and cause RTO to increase significantly.
When testing HA/DR scenarios, be sure to include the application. RTO for users will ultimately be based on application availability. Even if a database server is only down for 15 seconds, the user won’t care – they’ll view RTO by how long the application is unavailable.
Additionally, if your organization has contractual RTO obligations, ensure that all downtime related to an incident is accounted for – not just the hardware-related downtime!
Lost encryption keys and expired certificates
If any data is encrypted, do you know where the necessary encryption keys and certificates are backed up to? Are they backed up at all? Are certificates valid and not set to expire anytime soon? Will appropriate alerts fire when a certificate nears its expiration date?
Ensure that answers are available to all of these questions NOW! Mishaps with security during an emergency can turn a manageable problem into a nightmare of epic proportions.
The number of large organizations that experience crippling outages because a certificate expired, or an encryption key was lost, is a scarily big number!
Expired web domains, subscriptions, and licenses
Organizations rely on a wide variety of hardware and software solutions to power their own software, including:
- Web domains
- Hardware subscriptions (either on-premises or in the cloud)
- Software licenses for application, database, and monitoring tools
- AI/ML algorithms and solutions
- Email systems
- Hosting services
Staying on top of subscriptions and recurring costs requires effort by both technical and business personnel. Engineers need to be ready to update hardware and software whenever needed, ensuring that everything is working, patched, and that no service interruptions occur.
Additionally, someone in a finance role must ensure that the bills for these services are paid on-time, and that the paperwork needed for uninterrupted service is managed correctly. Yes, paperwork can cause downtime – I’ve even seen outages caused by expired credit card subscriptions!
Responders look for technical causes first, so business-side problems like these are easy to miss – and they can cause a real, unpredictable impact on RTO.
Staying on top of these responsibilities is therefore very important, even if subscriptions are configured to routinely auto-renew.
Cloud service retirements and vendor changes
What do our favorite cloud services all have in common? They are always changing!
Whether using Azure, AWS, Google Cloud or something else, their tech stacks evolve and change regularly. At any point in time, features may be deprecated, discontinued, changed, renamed, or moved.
While they do their best to communicate changes to us, ultimately it’s our own responsibility as users of their tech to stay on top of changes before they happen, and adjust processes when needed.
Most cloud software changes will not be breaking changes. Our goal, then, is to stay focused and pick out those that are more significant. For example, in 2026 alone, a total of 23 major Azure services were retired – as of time of writing (September 2026).
If you had software relying on them that was not updated to newer versions/apps, said software would have broken. That may have resulted in an outage – necessitating a speedy fix and leading to extended downtime and hassle.
The impact of discontinued/retired services
Additionally, while a discontinued service will often be allowed to be used for a grace period of time, new resources featured in the newer version may already be required – and using these newer apps/services may involve additional learning, configuration, and changes. All of that takes time and, therefore, impacts RTO.
Microsoft provides a reliable window into the timeline for retiring services in their documentation here.
The impact of vendor changes
Another notable event worth considering: are any software tools or services you used getting sold to another company? If so, is your organization happy with that change?
Do you anticipate problems that may stem from the new ownership, such as conflicts of interest, price increases, retirement, or changes in quality of service or customer service?
It pays to periodically check the status of tools you rely on. Make sure they’re well-maintained, and that any major announcements aren’t missed by you or your organization. Have alerts or notices sent to the correct people so that deprecation/discontinuation events are communicated effectively.
The impact of these events on RTO can be very unpredictable; they’re likely the ones that could cause the most trouble and require the most effort to resolve!
You may also be interested in…
What is cloud database vendor lock-in? The 3 types explained (and how to audit yours)
Skipped validation: recovery isn’t over when the restore finishes
One final area that has a very real impact on RTO is recovery. Consider this hypothetical outage sequence of events:
- Database server crashes (3 minutes).
- Root cause is identified as a storage subsystem failure (3 minutes).
- The database server is recovered from a server snapshot, but the database restores in a corrupt state (5 minutes).
- The new server is imaged, brought online, and restored from recent backups (15 minutes).
A total of 26 minutes were needed to follow a sequence of events from disaster to recovery. The missing step #5 is testing and validation of any applications that depend on that database server.
Does your organization have a reliable process that can quickly QA a recovered environment to ensure that the hardware, software, and user-facing applications are all performing adequately?
This step is often forgotten in the rush to restore services and minimize downtime, but there is confidence gained in knowing that a disaster will not persist for days or weeks as unforeseen issue arise. Most disaster-related problems are easiest to fix when first identified, rather than later.
After all, the last thing anyone wants in this situation is for an application to be handed back over to users who then discover lost data, broken functionality, or some other nightmare fuel!
But it could be even worse…
Now consider an alternate reality to the above sequence of events:
- Database server crashes (3 minutes).
- Root cause is identified as a storage subsystem failure (3 minutes).
- The database server is recovered from a server snapshot, but the database restores in a corrupt state (5 minutes).
- The new server is imaged, brought online, and restored from recent backups (15 minutes).
- The application is turned back on and users are allowed to log back in (2 minutes).
- Two hours later, users notice missing data.
- Database engineers discover that the key database was restored from incorrect backups. The transaction logs were never restored, resulting in 18 hours of data loss (15 minutes).
At this point in time, the situation becomes complicated. Data is now in a poor state, consisting of historical data from prior to the restore point and new data added by users since the application was restored.
Users have unexpectedly lost data but, because the database has been brought online for two hours of typical usage, there’s no straightforward way to restore it.
Making the call
Consequently, the organization is now forced to make a difficult decision between these options:
- Stop applications and perform the restore operation again, using the correct stopping point. The two hours of recent work will be lost.
- Do nothing and accept that 18 hours of history are lost.
- Restore a separate copy of the database to the correct point of time and assist users with the merging of lost data into their current environment, where needed.
None of these options are perfect. They result in data loss, extensive work for engineers, or both.
So, what exactly was the RTO related to this incident? It’s complex enough that the final number would be an estimate.
Does the application being up and running constitute an end to the outage, even if data was lost?
Does the outage persist until user data is made whole again?
Is the outage period customized based on customers that were affected and those that were not?
I personally want nothing to do with any of these decisions. Treat the QA after an outage as being just as critically important as the QA applied to new software features. The last thing anyone wants is a disaster immediately followed by a sequence of subsequent disasters.
These situations erode customer confidence and put significant strain on engineering teams and the leaders who must explain the fallout to customers.
Conclusion: how to build an RTO you can actually meet
Recovery Time Objective (RTO) is a key organizational metric that helps define an application’s availability and reliability.
Customers and users of an application will judge its quality based on the published RTO, as well as the RTO they observe while using the software.
However, determining a meaningful RTO, and conducting business in a way that honors that goal, is not easy. Many processes are required to ensure that maintenance and disasters are managed quickly, efficiently, and effectively.
Mistakes can lead to extended outages, lost data, and lost confidence. That’s why, while the time needed to plan ahead may seem significant, what it ultimately saves in time, money and stress, is greater.
In summary, here’s how to build an RTO you can actually meet:
| Stage | What adds to RTO | Question to ask |
|---|---|---|
| Detection | Metric sampling interval, missing alerts | How long until we know something’s wrong? |
| Response | Time for someone to sign on and start | Who’s on call, and what’s documented? |
| Restore | Backup size and count, IO, bandwidth | Have we timed a restore at our real scale? |
| Failover | HA switchover time | What can fail over automatically? |
| Application | Config, DNS, connection string changes | Are these scripted? |
| Security and licensing | Keys, certificates, subscriptions | What expires, and who gets alerted? |
| Validation | QA of the recovered environment | How do we confirm data and functionality are correct? |
Let me know your thoughts
A lot of very different tactics were presented in this article as ways to improve RTO and disaster response times. I hope at least a few are new and useful to you, and that you enjoyed reading.
Feel free to leave your thoughts down below – I’d love to hear from you!
Subscribe to the Simple Talk newsletter
FAQs
1. How do you calculate RTO?
Add up every step between failure and users working again: time to detect the problem (including the metric sampling interval), time for someone to respond, any failover or restore time, application reconfiguration such as connection strings and DNS, and testing and validation. Then check the estimate with a realistic recovery test. A figure based only on restore speed is usually too optimistic.
2. Why does recovery take longer than the RTO?
RTO targets usually count only hardware or restore time. In practice, recovery also includes detection delay, the time before someone starts responding, slow restores of large or numerous databases, manual application changes, expired certificates or lost encryption keys, and post-restore validation. Each step the estimate never included adds minutes or hours.
3. What is the difference between RTO and RPO?
RTO (Recovery Time Objective) is how long a system can be down before the impact is unacceptable. RPO (Recovery Point Objective) is how much data, measured in time, you can afford to lose. Restoring a database quickly but without its transaction logs can meet your RTO while badly missing your RPO.
4. Does a successful backup mean you can recover in time?
No! A successful backup job only shows the file was written. Restore it regularly in a scenario that mimics a real disaster, using the scripts you’d rely on in an emergency, and time it. Restore time for very large databases, or thousands of them, adds directly to RTO.
5. Does RTO include application downtime?
Yes, from the user’s perspective. A database server may be back in 15 seconds, but users judge RTO by when the application works again. Include connection string changes, DNS updates, metadata and monitoring updates, and application validation in your estimate, especially if you have contractual RTO obligations.
This document contains proprietary information and is protected by copyright law.
Copyright © 2026 Red Gate Software Limited. All rights reserved
Load comments